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ABSTRACT 

The three parts of this study concern the applicaticm 
of diagnostic testing to measure the effectiveness of classification 
training, the development of a systematic approach to applying the 
results # and a long-term study of employee retention of 
classification information. The measurement instrument selected for 
diagnostic testing of employees of a veterans Administration Hospital 
was a brief •True-Pa lse» test^ developed in 1968, with 20 items and 
called a Prototype Diagnostic Test. An alternate test, developed in 
1972, with 20 items and called a Classification Questionnaire, was 
used after completion of refresher training, and the results were 
compared yttth the diagnostic test score; thus it was possible to 
quantify the effectiveness of the refresher training. The test 
results were also used to evaluate Divisions and Services of the 
hospital • The long-range study of how long the average employee 
retained a satisfactory amount of classification information without 
refresher training utilized a randcm sample of 137 nursing assistants 
over a period of four years, after which they were tested with 
basically the same information, statistical analysis of the results 
showed that the average score of the population fell from 86% to 80%. 
It is concluded that use of a diagnostic test covering basic 
classification information is an effective way to determine 
classification training needs, provide a basis for measuring the 
effectiveness of classification training, identify employees for 
remedial training, and provide for comparative evaluation of the 
program. (For related document, see TM 002 335.) (DB) 
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l^j the effectiveness of classification training, development of a systematic 
appvoach to applying the results, and a long-term study of employee 
retention of classification Information. For detailed information about the 
conceptual frAaawork utilized in developing this study, see "Training 
. 3«lust:ion: A Guida to its I' binning, Deveiopi,v-it, and Use in Agency Training 
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A course evaluation or impression of a class recorded by a trainer at the 
conclusion of training does not appear to be a true evaluation. It is more 
like a miniature popularity contest. At best, it is so highly subjective 
that it appears to be unsuitable for measuring the amount of information 
imparted by the instructor. Ai\ achievement test is useful as far as 
it goes, but since ws don't know how much a class knew before the training, 
this kind of. test alone will not tell us how much the class improved as 

^Training Systems and Technology Series: No. IV. Bureau of Training, 
U. S. Civil Service Commission, May 1971, which is available from the 
superintendent of Documents, U. S. Government Printing Office, Washington, 
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a result of our training session. Knowledge of classification information 
is not apparent on-the-job unless the employee is submitting an appeal: 
therpfore. supervisory evaluations are also not an appropriate measuring 
device for this subject. Subjective tests are time consuming to evaluate 
and were rejected on the basis that quick, in-class feedback was an important 
goal. 

III. DEVELOPING A DIAGNOSTIC TEST 

Based on these considerations, we settled on a brief "True-False" test to be 
I P given without preceding training except what the employees were given in 

previous years. (This study excludes the initial training of new employees.) 
Following is a sample of the prototype test given in 1968: 



1968 PROTOTYPE DIAGNOSTIC TSST 
CLASSIFICATION QUESTIONNAIRE 

YOUR NAME 

YOUR DIVISION OR SERVICE 

TODAY'S DATE 



A. Answer the questions below by checking the word "YES" or "NO" in the 
space provided: 



1. I have a copy of my job description. 

2. My job description is accurate and up-to-date 



YES ( ) NO ( ) 
YES ( ) NO ( ) 



;B. Answer the following questions by drawing a circle around the word 
"TRUE" or "FALSE" after each question: 

1. If I want to know how my job was classified, I can go to the Personnel 
Division, talk to the Position Classifier and see the Classification Standards 
used to decide the grade of my job. 

TRUE FALSE 

2. My supervisor is available to answer questions I may have about my job 
or my grade, and I would usually talk to him first. 

TRUE FALSE 



3. If, after talks with my supervisor and the Position Classifier, I still 
balicve my grade should be hiphar, or classified in a different series, I do 
not have any further appeal rights. 



TRUE FALSE 



4. The niore work my supervisor gives me to do, the higher my grade should be, 
even if the added jobs are the same as work I have been doing all along, and 
the work is in my job description, 

TRUE FALSE 

5. If I am assigned a new kind of work to do, and I am going to keep this 
new work ec a regular part of my job, and the work is more important or more 
difficult than the work I have been doing, this new woik should be added to 
my job description. The Position Classifier should look at the job descrip- 
tion to see if my position is worth a higher grade, 

TRUE FALSE 

6. Everything I do on the job should be x^ritten down in my job description. 

TRUE FALSE 

7. I should be able to tell who my supervisor is from reading my job 
description. 

TRUE FALSE 

8. Classification standards show examples of jobs at different grades. The 
Position Classifier reads my position description anc" reads the classification 
standards for my job. Then he decides which example in the standard is most 
like my job, and assigns a grade. 

TRUE FALSE 

9. The same Classification standards are used all over the U.S. in Federal 
jobs like mine, so that an employee doing exactly the same job in Internal 
Revenue Service in Michigan as I am doing for the VA in Bedford, will have 
his position description classified with the same standards, will have the 
sarae grade and get the same pay as I do. This is one of the reasons we have 
classification standards. 

TRUE FALSE 

10. New Classification standards are written only by Personnel, 



TRUE FALSE 

11. My oxm service or division representatives help ;;rite classification stan- 
dards, I'or example, Supply people help vrite standifrds for Supply jobs, 
^^ursin^ people help write standards for Nursin3 jobs and so on. 

TRUE FALSE 

12. Once a classification standard has been written, it never changes, even 
if the work being done changes over the years. 

TRUE FALSE 

13. (GS employees only) The Civil Service Commission helps arrange for the 
study of new standards, writes the new standards in final form and then copies' 
are sent to all of our stations. The Position Classifier has to check all 



of the jobs at our station that are classified under the new standard. One 

of the things he checks is whether the graden of these jobs will change because 

the new standard is different than the old one. 



TRUE FALSE 



14. (GS employees only) The General Pay Schedule (GS) established under the 
Classification Act of 1949, (this is the schedule that tells how much you are 
paid, how often you get periodic increases and how much) is revised by 
Congress. For example, if everyone under the General Schedule gets a raise 
at the same tine, it is because Congress has approved a n*iw pay bill. 



TRUE FALSE 



15. (Wage Bc.rd employees only) The Veterans Administration Central Office 
arranges for the study of new classification standards, writes the new standard 
in final form and then copies arc sout to all of our stations. The Position 
Classifier has to check all of the jobs at our station that are classified 
under the new standard. One of tha things he checks is whether the grades 
of these jobs will change because the new standard is different than the 
old one. 



TRUE FALSE 



16. (Wage Board employees only) The pay scala for W.ge Board employees is 
reviewed every year by surveying what people in the same jobs are making in 
private industry in this local area (Boston area, for example.) The wages 
collected on the survey are then sent to VA Central Office where they are 
studied and a new wage scale issued. The purpose of the survey is to see that 
our Wage Board employees are paid about the same as people doing the same 
work in local private industry. 



TRUE FALSE 



17. (Wage Board employees only) If wages paid in local private industry 
generally go down , you would expect your pay scale and wages to go ug^. 

TRUE FALSE 

18. If I am not satisfied that my job is properly classified, after talking 
to my supervisor and Personnel, I may appeal to the Veterans Administration 
Central Office. 



TRUE FALSE 

19. If the Veterans Administration Central Office turns me down on my appeal, 
I may then further appeal to the Civil Service Commission. 

TRUE FALSE 

20. (GS employees only) If I appeal to the Civil Service Commission first 
(that is, without appealing to the VA), then I may not appeal the decision 
of the Civil Service Commission to the VA. 

TRUE YiLSZ 
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The purpose of the test was two- fold. One, to determine whether the 
employee had retained a satisfactory amount of current classification infor- 
mation, in which case he would not need to attend a class covering this 
information. Second, if the employee had not retained a satisfactory amount 
of information, the test was designed to provide a basis for measuring his 
improvement in subsequent training, as well as indicatting what information he 
did not know; and, therefore, what should be emphasized in subsequent training. 

Analysis of these goals will show certain subjective elements which must be 
noted. How much information is '^satisfactory?*' VJhat classification infor- 
mation is considered 'essential?^' The answers to these questions will vary 
widely, depending on a variety of factors including the intelligence and 
motivation of the population trained, the practical limits of training efforts, 
the relative importance of the subject to management, as well as variations 
in determinations of what is basic to the subject by different trainers. 
Once these decisions were made, however, development of the test was predicated 
on objective principles. 

^« The Questions Were Carefully Limited to the Essential 
Information Agreed Upon. 

The use of secondary information was avoided. For example, 
there are no questions about the history of the Class Act, 
which was considered nonessential, even though this might 
have been used as introductory material in the classification 
information course. Our impression, based on experience, is 
that most employees do not have a strong interest in the history 
of the Class Act, nor do we believe there is an essential 
"need to know.'* Basically, we reasoned, employees tend to 
retain only information which contains strong elements of 
self-interest or which are essential to functioning on-the-job. 

5* 



ERIC 



For example, employees tend to be very interested in their 
pay schedule and how it is determined. Employees need to know 
their appaal rights and how to obtain classification standards 
for review. These two underlined elements, then, became our 
criteria for what is essential. 

^' Th p. Test Was Validated in Several Wavs 
The test was given a dry run by artainistering it to saveral 
different populations, including our office staff, known experts, 
groups with diverse backgrounds and training. In this process 
we were successful in eliminating ambiguous questions and 
empirically establishing the validity of the test within 
reasonable limits for its use. 

c. An Alternate Test Was Daveloped 

To provide for retesting of the same population after remedial 
training, and for studies of long range decay of information 
remembered, it was necessary to develop an alternate test. 
The alternate test alters the sequence of presentation of 
questions and in some instances alters the wording of the 
original test. This is to prevent triggering remembered correct 
sequence of responses rather than remembered information. 
This test was validated in a similar manner to the first test. 
With ambiguous wording being grat^ually xreeded out. This was 

f 

facilitated by one-to-one discussions with office staff or 
experts to whom we had given the test. They frequently 
provided excellent feedback on questions liable to several 
different interpretations. Following is a sample alternate test 
O used during 1972: 

6. 



1972 ALTERNATE TEST - CLAnf^IFICATION QUESTIONNAIRE 



YOUR NAME 



YOUR DIVISION OR SERVICE 



TODAY'S DATE 



A. Ansver the questions below by circling the word "YES" or '*N0" as 
appllcab le: 



1. I have a copy of my position description. Yes No 

2. My position description is accurate and up-to-date. Yes No 

B. Answer the following questions by circling the word 'TRUE" or "FALSE'' 
as applicable: 



1. Everything I do on the job should be vnritnen into my position 
description. 

TRUE FALSE 

2. Any difficult or important work that I do regularly should be written 
in my position description, 

TRUE FALSE 

3. I should be able to tell who my supervisor zs from reading my position 
description. 

TRUE FALSE 

4. My supervisor gives me more work to do. This new work is just as 
difficult as my old work. As a result of the increased workload, I should 
be given- a higher grade. 

TRUE FALSE 

5. My supervisor assigns me work which is more complicated and difficult 
than I have been doing before. This work is going to become a regular 
part of my job. If my supervisor adds these new, more complex duties to 
my position description, the Position Classifier must review my position 
to see if the new duties will n^sult in a higher grade for my job. 

TRUE FALSE 

6. Classification Standards are secret. Only Classification Specialists 
can see them. 



TRUE FALSE 

7. My supervisor is available to answer questions I may have about my 
job or my grade, and I would usually talk to him first. 



TRUE FALSE 



8. If I still want to know more about how my job was classified, I can 
go to the Personnel Division, talk to the Position Classifier and see 
the Classification Standards used to decide the grade of my job. 

Q jc ^ , ^ TRUE FALSE 

•11 1 supervisors and the Position Classifier, I 

still believe my grade should be higher, or classified in a different 
series, I do not have any further appeal rights, 

TRUE FALSE 

e?,-ii^5^ after talking to my supervisor and the Position Classifier, I am 
still dissatisfied with the way my job has been classified, I may appeal 
to the Veterans Administration Central Office 

TRUE FALSE 

11. If the Veterans Administration Central Office turns down my appeal. 
I may then further appeal to the Civil Service Commission. 

TRUE FALSE 

12. (GS employees only) If I appeal to the Civil Service Commission first 
(that IS, without appealm to the VA) , then I may not appeal the decision 
of the Civil Service Conaaission to the VA. 

TRUE FALSE 

13. Classification Standards show examples of jobs at different grades. 
The Position Classifier reads my position description and reads the 
classification standard for my job. Then he decides which example in the 
standard is most like my job, and he assigns a grade to my job. 

TRUE FALSE 

14. New Classification Standards are written only by the Personnel Division. 

TRUE FALSE 

15. My own service or division representatives sometimes help write 
Classification Standards. For example. Supply people in Washington help 
write standards for Supply jobs. Nursing people help write standards for 
Nursing jobs, etc. Sometimes naw standards are sent to our hospital for 
review before they are published. 

TRUE FALSE 

16. Once a ClassificaMon Standard has been written it never changes, 
even if the work being done changes over the years. 

TRUE FALSE 

17. The Civil Service Commission arranges for the study of new or 
revised standards, writes the new or revised standards in final form, and 
then sends copies to all of our hospitals. The Position Classifier has 
to check all of the jobs at our hospital that are classified under the 



new or revised standards. One of the things he checks is whether the 
grades of these jobs will change because the new standard is different 
than the old one* 

TRUE FALSE 

18. (GS employees only) The same Classification Standards are used all 
over the United States in Federal jobs like mine. So, If I am a General 
Schedule (GS) employee, 1 will have the same grade as another GS employee 
in Michigan, if he does the same work that I do. 

TRUE FALSE 

19. (GS eiuployees only) The Department of Labor (Bureau of Labor 
Statistics) conducts a national salary survey of private industry to 
determine how such salaries compare with salaries paid us under the General 
Pay Schedule (GS) for comparable work. The President has to approve the 
nev; pay schedule they recoirrmond. 

TRUE FALSE 

20. (Wage Board employees only) If I am a CFWS C^/S, WL, WG) employee, 
and I have exactly the same job as another CFWS employee in the Boston 
Naval Shipyard, I would have the same grade and pay, since we are both 
working in the saip.e wage area. 

TRUE FALSE 

IV. THE TEST EHVIRONMEI^T 

With the exception of the long range study to be discussed later, in testing 
our employees we incorporated the principle of immediate reinforcement as 
much as possible, so that the testing situation became, in part, a learning 
situation as well. This v;as done by scoring the test at the test site 
immediately, returning to each employee his scored test, and going over the 
test, question by question. This usually produced animated discussion, 
immediate reinforceuient of correct information, and correction of inaccurate 
responses • Since considerable ego envolveraent is evident in a testing 
situation, it was important to emphasize the diagnostic nature of the test. 
Going over the test gives a feeling of finality to the session which helps 
disguise the selective nature of subsequent remedial training. No time limit 
was used because we found 10-15 minutes adequate for everyone to finish. 
For quick administration and rapid key scoring it was important to keep the 



test brief. 



V, SCORING 

Again the trainer will encounter the problem of subjectivity. How many in- 
correct answers are "too many?" Should any of the questions be "lock-out" 
questions? In other words, are there any questions vhxch are considered 
mandatory for employees to answer correctly for minimum acceptable level of 
knowledge of classification information? As before, the answer will differ 
from one trainer to the next. We used cutting scores varying from 70 :o 
73%, depending on the scoring spread in different groups, with the questions 
related to availablity of standards and appeal rights as "lock-out" queftions, 
Any employee who missed a lock-out question or scored lower than the cutting 
point for his group was scheduled for refresher training. 

VI. USE OF TESTING RESULTS 

It was apparent from initial results that using a diagnostic tes: was going 
to improve training effectiveness significantly. We had narrowed our 
audience, if not to those x^ho would automatically be interested, at least 
to those who needed the infomiation. Further we laiew precisely what infor- 
mation they needed. Conversely, wa had eliminated the presence of employees 
on whom our efforts would be wasted, and who would tend to react negatively 
because they already knew all the ansv/ers. 

Further, the test results would serve as a basis for quantifying the exist- 
ing knowledge about classification. This data in turn could be used to rate 
classification knowledge objectively by Division and Service. In addition, 
if we used the test after refresher training, we could quantify and demon- 
strate improvement related to the training. 

a« Classification Refresher Training 

Although we were careful to explain the nature cf the diagnostic 



test, we were also careful not to tell employees how ue in- 
tended to use it. Nor did we identify the refresher training 
as "make-up" or "remedial training." We were careful not 
to make the employees scheduled for refresher training feel 
"singled out" in any r.;ay. One of the ways we avoided this 
stigma was to delay scheduling refresher training until several 
months after the diagnostic testing was compieted in order to 
avoid the two events being connected. Another technique in- 
volved mi::ing employees from different Divisions and Services, 
and scheduling different employees from a given Division at 
several different sessions. This was accomplished automati- 
cally because the classes were co^aposed on che basis of similar 
questions missed. This tended to make very inconspicuous the 
employees who passed the diagnostic test and who did not 
attend. 

In two experimental situations at the end of our study we did 
announce the purpose of the test and the selective nature of 
the refresher training to two small groups. We concluded 
from subjective observations of class reactions that, although 
this approach is feasible to use and probably increases 
motivation to score high on the diagnostic test, it hao a 
strong negative effect on subsequent refresher training and 
has other undesirable psychological effects if the "selecting 
out" aspects are emphasized. We have not used it since this 
trial . 

Our last method of "smoke-screeninr" the remedial nature of 
refresher training: Although the refresher material was 
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geared to the needs of the specific group based on the results 
of their diagnostic test, this information was sandwiched be- 
tween Uim-^.lv additional subject matte r - for example, "how 
the new annual October review c ,a. ..orlcs." If the 

audience constitutes cms employees, the topic might be 
"The new 5 step nonsupervisory CTJS wage schedule." 

b. IJsln?, ?hc Test to T^v ^iluate Individual Employee Progress 
At the coticlusion of refresher training, employees were re- 
quired to complete an alternate form of the test. By comparing 
the results with the diagnostic test score taken previously, 
it is possible to quantify the effectiveness of the training 
for each employee. Note the significant improvement in this 
comparison: 

RESULTS OF FIRST SESSIC\' 
SELECIED CLASSIFICATION REFRESHeI! TI^INIMff 
NURSING SERVICE 



Name (Altered) 


Results of 
Diagnostic 
Test FY '68 


Results of Re- 
test Following 
Refresher Training 


Si'ilTI-I 




50.5 


100 


JCMSS 




67 


90 


DOE 




72.5 


85 


SKiART 




50.5 


80 


KILROY 




78 


80 


EDWARDS 




67 


80 


DAY 




72.5 


65 


BRODIE 




61.5 


55 




This 


comparison was attached 


to a memorandum to the Service 



concerned, emphasizing the new approach: 
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TO : Chief, Nursing Service (118) 



' J' Chief, Personnel Division (135) 

SUBJ: Interim Results of Classification Remedial Training 

1. Eight nursing astiistants were given a one-hour refresher training class 
on Classification Information June 26, 19 . We have attached the results. 

2. Wc believe you will agree that this group of employees was on the v^ole 
dramatically improved. We further believe that the results demonstrate that 
a diagnostic testing approach to classification training is effective. We 
plan to nvalce minor revisions to the clasisif ication questionnaire, coraplete 
diagnostic testing of those nursing assistants v;ho have completed initial 
orientation and have not yet been tested, following this operation we 
\d.ll furnish you a memorandum of results and a revised service average. 



GEORGE F. FLAI^GAN 

Att. 

Obviously we did not comment about the two employees whose 
scores went down instead of up I We were intrigued by these 
two scores and decided to wrk out a third procedure to follow 
up on such cases. We scheduled routine desk audits \d.th these 
two employees which included another review of classification 
information. Several oonths after the refresher training we 
retested them with the original test. The results were mixed. 
One of the employees scored 80 and the other scored 55. We 
believe the trainer must draw the line somewhere, and we 
accordingly did not pursue this employee who scored 55 any 
further, esccept to make certain he was scheduled for refresher 
training the follov/ing year. (With similar results, we mi^t 
add.) It is interesting to note that simple documentation of 
these procedures provides the trainer with rather clear evi- 
dence that if an employee is deficient in classification 

Icnowledge, it is not because of the lack of reasonable effort 
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by the trainer to make him otherwise. 

c. Ufling the Tes t Results to Evaluate Divisions and Services 
Using a simple statistical summary of results by Division, 
and subjectively determining adjective ratings that correspond 
to score ranges, it was easy to evaluate Divisions against 
this standard. v?e used the follov/ing adjective ratings and 
ranges: 

0-69% Unsatisfactory 

70-79% Satisfactory 

00-89% Higli Satisfactory 

90-100% Outstanding 
Samples of summaries furnished to Ilursing Service and Supply 
Divi-iion follow: 

NURSING SERVICE SUMi-l^RY 

EMPLOYEE QUESTIONNAIRES 
Date Tested 1/9 - 2/2, 19 
SUMMARY 



36 


employees 


at 


100 


3600 


38 


II 


at 


94.5 


3591 


52 


II 


at 


89 


4628 


41 


•1 


at 


83.5 


3423.5 


21 


II 


at 


78 


1638 


21 


II 


at 


72.5 


1522.5 


7 


11 


at 


67 


469 


4 


u 

• 


at 


cl.5 


240 


1 


II 


at 


56 


56 


3 


II 


at 


50.5 


151.5 


2 


II 


at 


45 


90 



NURSING SERVICE SUMMARY CONTINUED 

^ " at 39.5 - 39.5 



227 

19,455.0 

OVERALL RATING: 85.7 
EMPLOYEES REQUIRING REFRESIiER TRAINING 

(List of 18 employees) 



SUPPLY DIVISION SUSyt'JARY 

EMPLOYEE QUESTIONNAIRE 
Date Tested June 13, 19 
SUMMARY 



6 


employees 


at 


100 


1 


II 


at 


95 


3 


II 


at 


94.5 


1 


II 


at 


89 


1 


II 


at 


85 


1 


ti 


at 


83.5 


4 


II 


at 


70 



600 
95 
283.5 
89 
85 
J3.5 
312 



1 employee transferred 

OVERALL RATING: 91.0 (outstanding) 
EMPLOYEES NOT TF.fiTPn 

^ COYLE — (leaving Friday) PCS 
WYSNEWSKI — (annual leave) 

15 19 FT 6/13/ 

EMPLOYEES FOR PROGRAI.£.IED REFRESHER TRAINING; 
None 



The Nursing Service S^r, above ™s l,.portant to our study 
because It Indicated that our questionnaire getting a good 
■■spread., of scores in a relatively large previously trained 
population, as well as that the number of employees below the 
■■cutting., score of 72.5 ™s about „hat w ™,ld expect en^erlcally 
from this group on the basis of previous experience. The high 
-ores in Supply Division suggest a different course content 
.=By bo in order for this group. Refresher training be 
"orthwhile for the four employees who scored 7S. This depends 
« the Objective of the trainer. In the first year these so.- 
manes wre attached to a «e,norandu„ report to each Division 
and Service at the time of a„„ual classification review, m 
the following year we were able to furnish comparitive date 
on the Division standings and get a new sense of direction for 
our training program. Following are two Supply Division sum- 
maries for that year and cover .nemorandum. of interest is 
the method of adjusting the previous year's raw score in Section 
I of the summary. This approach is only one of many that could 
be used. 

CLASSIFICATION INFORMATION PROGSAi.i, py 



DIVISION/ SERVICE: Supply 

DATE, TRAINING & TESTING: June 10-24, 19 

I. DIAGNOSTIC TESTING BASE: (last FY o 

transferred, separated or retesied aftL I J^f* ™^ °^ employees 

employees transferred in) "''^^'•^^ remedial training, plus scores of 

Number of employees 13 Adjusted Raw Score 1246.5 

II. RESULTS OF BASIC TRAINING/ REFRESHER TRAINING: 

5 employees at ICQ _ 590 
— 3 — employees at 89 - 267 

® 767 16. 



III, FY SUMi'IARY 

This FY: 21 employees at 2013.5 - 95.8 a verage (Outstanding) 

91.0 (Outstanding) 
Improvement in average 4.0 

IV. EMPLOYEES NOT TESTED: 

(None) 
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CLASSIFIMON INFOK-iATION TRAINING - SUPPLY DIVISION 
INDIVIDU/VJ. srORRR LAg^ 

I. 



II. 



IV. 



THIS FY 



DIAGNOSTIC TESTING 


SCORE 


SCORE 


DOE, John 




78 


BASIC COURSE NEW Ei>iP. 






JAMES, Henry 




100 


COPPERFIELD, Drvid 




100 


SMITH, John 




100 


SELECTIVE REFRESHER TPAINING 




SWIFT, Johnathain 


83.5 


100 


TQLiaEN, R. R. 


78 


89 


BROtJN, Tom 


78 


89 


aEi>ffiNS, Samuel 


78 


89 


JONES, John 


78* 


100 


TRANSFER IN 






HARP, Uriah 


94.5 




Chief, Supply Division (134) 
Chief, Personnel Division(135) 


Date: June 26, 19 
(135)HWH:eml 


Annual Review 







ledge. A Su«nary ofsuppS 01^1^™%°! ,^ cUsstflaitlon Infometlon know- 
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2. Tliis year, we are confining classification training to the following: 

a. Basic Classification course for new employees. 

b. Training for those employees who evidenced a need for such training, 
or brush-up in specific mandatory areas, such as appeal rights and 
availability of standards based on last years test results. 

c. Special course for Wage Board employees to inform them about the new 
Coordinated Federal Wage Program, which will be implemented soon. 

3. With the exception of the Wage Board course, which will be given to your 
wage board employees sometime next month, we have summarized classification 
training results on the attached sheets. We do not plan any testing for the 
wage board course. 

4. Your new rating for this FY is 95.8 (Outstanding), an improvement of 
4.C over last year. 



GEORGE F. FLANAGAW 



Attach: (2) 



On the individual score sheet, note that the test was used to 
evaluate the results of the basic course for new employees also, 
and that this data becomes part of the Division's score. 

d> Station Comparative Evaluations 

Once the station has gone thru a cycle of testing, it was a 
simple matter to summarise the Division and Service results 
and come up with a station rating. 
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CLASSIFICATION INFORl»iATIOiI TEST - STATION SlE'tlARY FY 



DIVISION/ SERVICE 


Empl . 
Tested 


Raw 
Score 


Average 
Score 


Adjective 
Ratine 


# for 
Tralnlne 


Not 
Jest 


SUPPLY 


17 


1548 


91.0 


Outst. 


0 


2 


DENTAL 


8 


732 


97.7 


Ovtst. 


*l 


0 


RADIOLOGY 


3 


256 


05.3 


HS 


Q 


0 


CHAPLAINS 


5 


488 


97.6 


Outst. 


0 


0 


FM£eR 


34 


3108.5 


91.4 


Outst. 


0 


5 


NURSING 


227 


19455.0 


85.7 


HS 


19 


45 





1 O 


1150 


88.5 


HS 


1 


0 




7o 


66S5 


07.6 


if5 


8 


18 


lAB ORATORY Q\7r 




373,5 


74.7 


Sat. 


2 


0 ** 


Dlj XA7 • 1 JcVi li\Lr£ii i£ilN 1 


70 


5930,5 


75.1 


Sat. 


37 


10 




7 


667 


35.2 


Outst. 


0 




I/iEDT PAT A nWT M 


GO 


7142 


09.3 


HS 


1 


15 


DIETETTP 


loo 

123 


10306 


o o 

U J ,U 


HS 


11 


10 




15 


1236 


02,4 


HS 


0 


3 




i\J 


1004,5 


94,2 


Outst. 


0 


0 **** 


FISCAL 


y 


fie 

856 


95.1 


Outst. 


0 


0 


CLINICAL PSY 


o 


coo C 

583,5 


97,3 


Outst. 


0 


1 


COUt^SELING PSY, 


15 


1362.5 




Outst. 


0 




PTIA RivIA PV 


4 


372.5 


93.1 


Outst. 


0 


0 


DIRECTOR & COS 


15 


1406.5 


93.0 


Outst. 


0 


0 


SURGICAL 


3 


261.5 


G7.2 


HS 


0 


0 


I-IEDICAL 


2 


183.5 


91.7 


Outst. 


0 


0 


STA.TIOK 






09.122 








755 


65958.0 


06.2196 


IIS 


80 


129 



^Secretary 

*****Re tested May 19 



**** 5 Supervisors tested but not recorded (all Outst.) 
_ '^^special effort needed here Outstanding 11 

**reconimend refresher training for High Sat. 10 



Based on this summary, the trainer can say, "we have a high 
satisfactory classification information program, but we need 
to concentrate on Building Management Division and Laboratory 
Service to raise their level of understanding." The summary 
also provides additional valuable information; i.e., how many 
people require refresher training? How many have not been 
tested? etc. The results also strongly suggest where supervisory 
training may be desirable, and pra^Juce a basis of discussion with 
the Chief of the Division. 
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Hiere are other station sundries which may prove useful in the 
classification review program, or to pinpoint lack of critical 
information. One such summary gives the results of Section A 
of the test which asks whether the employee has a copy of his 
position description, and whether it is accurate. In addition, 
this summary gives the results of the classifier's follow-up 
Of problems pinpointed, it may show trends over a two year 
span that alert the classifier where to concentrate position 
description reviex; efforts. 

CUSSinCTIOU ™*TI«, TEST - sr.«a, su.*rv FY- (p.a..po.ltion description 



DIVISIOiT/SSRVICE 


Emp. 

Tested 


Stated 
no 13..: 


* X havinj^ 

• p .cI . 


Stated 
!xl inacc . 


Proved Adden./ % p.d 

AC(2. J^aAna 


Supply 


17 


1 


94% 


1 


0 


1 


94% 


Dental 


8 


0 


100% 


0 


0 


0 


100% 


Radiol opv 


3 


0 


100% 


0 


0 


0 


100% 


Chaplains 


5 


0 


100% 


0 


0 


0 


100% 


PM&R 


34 


4 


88% 


7 


1 


6 


82% 


Nursing 


227 


26 


88% 


16 


*** 
14 


2 


99% 


Research 


13 


0 


100% 


0 


0 


0 


lC»/7, 


Engineering " 


76 


7 


91% 


4 


3 


1 


99^; 


Laboratory 


5 


0 


100% 


0 


0 


0 


100% 


BMD 


7G 


6 


92% 


2 


1 


1 


99% 


Personnel 


7 


0 


100% 


1 


1 


0 


100% 


mj) 


00 


14 


85% 


15 


8 


7 


88% 


Dietetic 


123 


17 


86% 


2 


2 


0 


100% 


NHCU 


15 


0 


100% 


0 


0 


0 


100% 


Social Work 


20 


0 


100% 


0 


0 


0 


100% 


Fiscal 


9 


0 


100% 


0 


0 


0 


100% 


Clin, P$y. 


6 


0 


100% 


0 


0 


0 


100% 
20. 



Coun, Psy, 

Pharmacy 
Dir./COS 
Surgical 
Medical 
TOTAL smnoN 



15 
4 

15 
3 

2 

765 



0 
0 
0 
0 
0 
75 



100% 
100% 
100% 
100% 
100% 
90% 



3 
0 
2 
0 
0 
53 



** 
0 
0 
0 
0 

30 



3 
0 
2 

. 0 
0 

23 



100% 
100% 

07% 
100% 
100% 

97% 



«^"-^P"-= or addenduu^ were pending 

rSSlorL:Sr:Lne^S.°' .naer^tanding of special ^Ul-Jo. 

Following is a station suimnary on the basis of the "loclc-out" 
questions. H^ere is the expected very close corr3lation between 
those Divisions with low rating:, on the test overall and those 
low on the -lock-out" questions, but sorae additional low 
Divisions suggest additional areas for improvement. The ques- 
tion numbers are related to the 1960 prototype test. 
CUSSIFICATION INFORMATION TEST - STATION SUHMARY 

E^p" ^W^^'r' '''"''I^^^ . (Appeal Rights) 

mmommcE e d ^ .^o^ ^ 

Supply 

Dental 
Radiology 
Chap lains 
PM&R 
Nursing 
Research 
Engineering 



17 


16 


94% 


Out. 


45 


90% 


Out. 


8 


8 


100% 


Out. 


24 


100% 


Out, 


3 


3 


100% 


Out. 


7 


78% 


Sat.* 


5 


5 


100% 


Out. 


13 


87% 


H.Sat. 


34 


34 


100% 


Out. 


99 


97% 


Out. 


227 


221 


97% 


Out. 


624 


92% 


Out. 


13 


13 


100% 


Out. 


34 


87% 


H.Sat. 


76 


74 


97% 


Out. 


197 


86% 


H.Sat. 
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c 
J 


5 


100% 


Out. 


10 


67% 


Unsat .* 


Bid?. Momt 


/o 


76 


977o 


Out. 


179 


76% 


Sat.* 


PprQnnno 1 


1 


7 


100% 


Out. 


21 


100% 


Out. 


i lC7 U • nUUlJLIi • 




79 


99% 


Out. 


191 


80% 


H.Sat. 




loo 

123 


122 


99% 


Out. 


312 


85% 


H.Sat. 




15 


15 


100% 


Out. 


30 


67% 


Unsat .* 




20 


20 


100% 


Out. 


58 


97% 


Out. 


Fiscal 




9 


100% 


Out. 


25 


93% 


Out. 


Clin. Psy. 


6 


6 


100% 


Out . 


18 


100%. 




Coun. Psy. 


15 


15 


100% 


Out. 


42 


93% 


Out. 


Pharmacy 


4 


4 


100% 


Out. 


12 


1007o 


Out. 


Dir. 6c COS 


15 


15 


100% 


Out. 


43 


94% 


Out. 


Surgical 


3 


3 


100% 


Out. 


9 


100% 


Out. 


Medical 


2 


2 


100% 


Out. 


6 


100% 


Out.. 


STATION TOTAL 


765 


752 


98% 


Out. 


1999 


87% 


H.Sat. 



*Remedial training to be given dviring next annual review. 



VII. OTHER USES CON SIDEREn 

The feasibility of loading personnel program review questions in addition 
to those in Section A of the test, was considered and rejected. The number 
of additional questions required and the subjective nature of much of this 
kind of material would make the test cumbersome and time consuming. A 
regular, separate attitude survey is considered a more satisfactory solution 
to overall personnel program review even if some duplication of effort is 
involved. 
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VIII. 


RETENTION OF CLASf?TPTrATTnM 


INFORMATTnn 


The use 


of 


this short test raised many questions for the trainer, but perhaps 


the mos 


t significant 


was, "How 


long will the average employee retain a 


satisfactory amount 


of classification information without refresher training?" 


To attempt 


to answer 


this question, 


a long-range study was conceived. It was 


decided 


to 


omit all 


classification 


training for a random selection of some 


200 nursing 


assistants for a period 


of four years and then retest them with 


basically the same information. 


The 


purpose was to find out how much their 


scores ' 


'decayed." A 


statistical analysis of the results is given below: 


STATISTICAL SIM-IARY OF U vfat? pTIW 


- CLASSIFICATION INF0RT4ATTON RPTirMTTOM 


EMPLOYEE NUMBER 


'68 Test 
Score 


72 Test Deviation Deviation Deviation 
S^^^e _(-) (0) M Rant 


1 






100 


100 


0 1.0 


2 






94.5 


100 


5.5 ■J-43.0 


3 






89 


100 


11 


4 






72.5 


100 


27.5 +119.0 


5 






100 


95 


■5 41.5 


6 






100 


95 


41.5 


7 






94.5 


95 


.5 +2.5 


8 






94.5 


95 


.5 +2.5 


9 






89 


95 


6 +44.0 


10 






78 


95 


17 +95.0 


11 






67 


95 


28 +120.0 


12 






45 


95 


50 +133.5 


13 






100 


90 


-10 71.5 


14 






100 


90 


-10 71.5 


15 






100 


90 


-10 71.5 
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16 


100 


on 


-10 




71.5 


17 ' 




yo 


- 4.5 




37.5 


18 




90 


-4.5 




37.5 


19 




90 


-4.5 




37.5 


20 




90 


-4.5 




37.5 


21 




90 


-4.5 




37.5 


22 




90 


-4.5 




37.5 


23 




90 






+6 


24 




90 






+6 


25 




90 






+6 


26 




on 






+6 


27 




QO 






+6 


28 


83 S 


on 




6.5 


+46.5 


29 




on 




6.5 


+46.5 


30 


O J • J 


on 




6.5 


+46.5 


31 


O J • 3 


on 




6.5 




32 


/o 


on 
90 




12 


+74.0 


33 


79 Q 
/Z.J 


on 
90 




17.5 


+97.0 


34 




oe 


-15 




91.5 


35 


1 nn 


oe 


-15 




91.5 


36 


1 nn 


o5 


-15 




91.5 


37 


100 


85 


-15 




91.5 


38 


100 


85 


-15 




91.5 


39 


94.5 


85 


- 9.5 




67.0 


40 


94.5 


85 


- 9.5 




67.0 


41 


94.5 


85 


- 9.5 




67.0 


42 


94.5 


85 


- 9.5 




67.0 
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94.5 


44 


89 


45 


fi9 


46 


89 


47 


89 


48 


83.5 


49 


83.5 


50 


83.5 


51 


78 


52 


72.5 


53 


72.5 


54 


65 


55 


56 


56 


100 


57 


100 


58 


100 


59 


100 


60 


100 


61 


100 


62 


100 


63 


94.5 


64 


94.5 


65 




66 


67 


67 


89 


68 


89 


69 


89 


70 


89 



ERIC 



85 


-9.5 




67.0 


85 


-4 




32.5 


85 


-4 




32.5 


85 


-4 




32.5 


85 


-4 




32.5 


85 




+1.5 


10 


85 




+1.5 


10 


85 




+1.5 


10 


85 




+7 


49 


85 




+12.5 


76.0 


85 




+12.5 


76.0 


85 




+20 


109.0 


85 




+29 


122.5 


80 


-20 




109.0 


80 


-20 




109.0 


80 


-20 




109.0 


80 


-20 




109.0 


80 


-20 




109.0 


80 


-20 




109.0 


80 


-20 




109.0 


80 


-14.5 




87.0 


80 


-14.5 




87.0 




« 1 /• ^ 




87.0 


80 




+13 


78.5 


80 


-9 




62 


80 


-9 




62 


80 


-9 




62 


80 


-9 




62 



erJc 



71 


89 


80 


-9 




62.0 


72 


83.5 


80 


'3.5 




25.5 


73 


83.5 


80 


-3.5 




25.5 


74 


83.5 


80 


•3 5 




25.5 


75 


83.5 


80 






25.5 


76 


83.5 


80 


•J •J 




25.5 


77 


83.5 


80 


•J • J 




25.5 


78 


83.5 


80 






25.5 


79 


83.5 


80 






25.5 


80 


78 


80 




:*2 


13.5 


81 


78 


80 




{•2 


13.5 


82 


78 


80 




1 o 

-3*2 


13.5 


83 


78 


80 






13.5 


84 


72.5 


80 




»•/ .J 


51.9 


85 


72.5 


80 




t7.5 


51.0 


86 


67 


80 




+13 


78.5 


87 


100 


75 


-25 




118.0 


88 


94.5 


75 


-19.5 




103.5 


89 


94.5 


75 


*19.5 




103.5 


90 


89 


75 


-14 




84.0 


91 


89 


75 


-14 




84.0 


92 


89 


75 


-14 




84.0 


93 


83^5 


75 






56.0 


94 


83.5 


75 


«8 5 




56.0 


95 


A*) S 


7c 
iJ 


-8.5 




56.0 


96 


33.5 


75 


-8.5 




56.0 


97 


83.5 


75 


-8.5 




56.0 


98 


50.5 


75 




+24.5 


115.0 


99 


83.5 


75 


-8.5 




56.0 
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100 


78 


75 


-3 




18 


101 


78 


75 


-3 




18 


102 


78 


75 


-3 




18 


103 


72.5 


75 




-I-3.5 


25.5 


104 


72.5 


75 




1-3.5 


25.5 


10!) 


100 


70 


-30 




126.0 


106 


100 


70 


-30 




126.0 


107 


100 


70 


-30 




126.0 


108 


94.5 


70 


-24.5 




115.0 


109 


94.5 


70 


-24.5 




115.0 


110 


94.5 


70 


-24.5 




115.0 


111 


94.5 


70 


-24.5 




115.0 


112 


89 


70 


-19 




101.0 


113 


89 


70 


-19 




101.0 


114 


89 


70 


-19 




101.0 


115 


83.5 


70 


-13.5 




81.0 


116 


83.5 


70 


-13.5 




81.0 


117 


72.5 


70 


-2.5 






118 


83.5 


70 


-13.5 




81.0 


119 


67 


70 




1-3 


18.0 


120 


67 


70 




+3 


18.0 


121 


61.5 


70 




+8.5 


56.0 


122 


39.5 


70 




J-30.5 


128.0 


123 


100 


65 


-35 




132.0 


124 


94.5 


65 


-29.5 




124.0 


125 


80 


65 


-15 




91.5 


126 


72.5 


65 


-7.5 




51.0 


127 


94.5 


60 


-34.5 




130.0 



f 



128 

129 

130 

131 

132 

133 

134 

135 

136 

137 

TOTALS 
AVERAGES: 



•94.5 
94.5 
89 
83.5 

78 

72.5 
83.5 
72.5 
72.5 

ICO 



60 

60 

60 

60 

£0 

60 

55 

55 

55 

50 



-34.5 

-34.5 

-29 

-23.5 

-18 

-12.5 

-28.5 

-17.5 

-17.5 

-50 



130.0 
130.0 
122.5 

99.0 
76.0 

121.0 
97.0 
97.0 

133.5 



11302.5 10905.0 
Employees 137 

86% 80% 



1313.0 



93 



417.5 



43 
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The population, due to turnover, was reduced to 137 during the 4 year .-de- 
cay period. Of the 137 employees, only 15 had scores which fell below the 

acceptable level of 70 during the period or about 117 Th. 

ttuuut ii/o. xne average score of 

the populat... un 0« eo 8«. In oehe. „«ds. eUs group wuia have 
the ea^ aajectlve rating as prior to the test. It U U.el, tha~ another 
group coul. conceivably fall ,y as ^ch as «,e adjective rating. So„e in- 
teresting, u academic, results „ere observed. Porty-three employees im- 
proved their scores on their o„, £or reasons undetennined. Eleven o£ 
these eoved fr» unsatisfactory to satisfactory ratings or better. This 
accounts in part for the smll drop in overall rating. 

It is suggested by this study that the amount of infor«tion loss is such 
that a number of alternatives may be considered: 

(1) detesting every four years is possible for an average program. 

(2) It may be considered desirable to reduce the information loss 



thru a two year retesting and selective training cycle. The fre- 
quency of retesting should be weiglied against the effect on em- 
ployee interest. 

(3) It may very well be that the frequency of retesting is best 
dictated by the adjective rating of the population. In other 
words, wait four years to retest an outstanding group, retest a 
satisfactory group every year. 

IX. coNgjiiniONS 

We conclude that the systematic use of a diagnostic test covering basic 
classification information is an effective way to (1) quantitatively and 
selectively determine classification training needs (2) provide a basis 
for measuring the effectiveness of classification training given on an 
individual and collective basis (3) identify employees for selective re- 
medial training and (4) provide a basis for comparative evaluation of this 
program. 

We further conclude that diagnostic testing improved the quality and 
effectiveness of this classification training program and provided a 
uasis for selecting future classification :ourse material more effectively. 
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